1. Field of the Invention
The invention relates to an image processing apparatus, image processing method, program and recording medium that are suitably applied, for example, when an image captured in a wide angle of view is projected onto a screen.
2. Description of the Related Art
In an existing art, in order to capture a landscape, or the like, in a wide angle of view, for example, a panoramic imaging method, a multi-camera imaging method in which a plurality of cameras are placed at a plurality of locations, an imaging method in which a low-resolution dynamic image is synthesized with a high-resolution static image, a multi-focus imaging method that uses a camera having a plurality of foci, and the like, are employed. Hereinafter, the above imaging methods will be simply described.
In the panoramic imaging method, images captured by a plurality of cameras are pieced together to generate a wide-area image. The panoramic imaging method includes a single eyepoint method and a multiple eyepoint method. In the single eyepoint method, because a plurality of cameras capture a subject at one and the same location, a pieced image has no boundary. In the multiple eyepoint method, because a plurality of cameras capture a subject at different locations, a pieced image has a boundary.
In the case of the single eyepoint method, the peripheral portions of images before being pieced deform because of the characteristics, or the like, of the lenses assembled to the cameras. Deformation of images causes degradation of a pieced image. If a deformed image is directly projected onto a screen, deformation in the image at the edge of the screen becomes further conspicuous. In addition, a common portion at which adjacent images overlap occurs around a boundary of images captured by the adjacent cameras. Here, the common portion of the adjacent images will be described with reference to FIG. 27.
FIG. 27 is an example of an imaging method using cameras that are arranged so that imaging directions of the cameras meet at one point. In this method, three cameras 101a to 101c are arranged in extended lines in the imaging directions with respect to a virtual focus 102 as a reference. Note that the imaging directions respectively coincide with the optical axes of the cameras. Then, a point at which the plurality of optical axes meet one another is used as a virtual “eyepoint of a camera array (a plurality of cameras)”. When images captured by the plurality of cameras are synthesized, it may be necessary to consider two types of eyepoints, that is, the “eyepoint of the camera array” and the “eyepoints of the cameras”. Here, the method for making the “eyepoints of the cameras” coincide with one another has two types, and these methods are called “single eyepoint method”. The first method captures a subject using a relay optical system with a physically one lens. The second method captures a subject so that cameras are respectively set to angles of view without any common portion at which adjacent images overlap. Even when the “eyepoints of the cameras” are intended to coincide with one another using a method other than the above two types of single eyepoint methods, it is difficult because the size of the lens differs among the cameras.
Referring back to FIG. 27, the cameras 101a to 101c each capture a subject located in a long-distance view 103 and a short-distance view 105 at the same angle of view. Then, the cameras 101a to 101c focus on an object focal plane 104. At this time, there is a common portion 103a at which imaging portions of the adjacent cameras 101a and 101b overlap in the long-distance view 103. Similarly, there is a common portion 103b at which imaging portions of the adjacent cameras 101b and 101c overlap is present in the long-distance view 103. In addition, there is a common portion 104a at which imaging portions of the adjacent cameras 101a and 101b overlap in the object focal plane 104. Similarly, there is a common portion 104b at which imaging portions of the adjacent cameras 101b and 101c overlap in the object focal plane 104.
The common portions 103a, 103b, 104a and 104b of the images captured by the cameras 101a to 101c are mixed in chromaticness pixel by pixel. However, when the eyepoints of the plurality of cameras do not coincide with one another, an object distance from each camera to a subject varies. A plurality of images captured on a specific focal plane (in this example, the plane of the object focal plane 104) may be pieced smoothly, whereas, when an object distance from each camera to a subject varies (in this example, a state where the long-distance view 103 and the short-distance view 105 are mixed), a joint between the images tends to appear to be unnatural (also referred to as “short-distance view split”). Thus, even when the common portions of the plurality of images are mixed with each other, it is difficult to maintain the images at high resolution.
The short-distance view split is described, for example, in Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2005-522108. Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2005-522108 describes a technique that an upgrading function that is learned in high-grade representation and low-grade representation is used to improve the grades of image data, video data and audio data.
On the other hand, in the multiple eyepoint method, cameras that can obtain images of the same resolution and chromaticness are arranged at a plurality of locations to capture a subject. An individual difference of each camera appears in a difference in zoom rate, or the like, so the influence of the individual difference is excluded to equalize the performances of the cameras. At this time, in order to capture a subject using cameras of which the characteristics are virtually equalized, an image obtained by each camera is calibrated by various methods.
Japanese Unexamined Patent Application Publication No. 7-143439 describes an image capturing apparatus. The image capturing apparatus integrates a dynamic image capturing unit that captures a dynamic image at a low resolution and a static image capturing unit that captures a static image at a high resolution, and obtains a high-quality image at a desired photo opportunity.
Japanese Unexamined Patent Application Publication No. 2005-318548 describes a technique for imaging so that a first image data sequence (image data sequence of low resolution and high frame rate) and a second image data sequence (image data sequence of high resolution and low frame rate) fall within the same range. When these two image data sequences are integrated to execute image processing, a high-resolution and high-frame rate image may be obtained.
The methods described in Japanese Unexamined Patent Application Publication No. 7-143439 and Japanese Unexamined patent Application Publication No. 2005-318548 assign time and resolution for each camera. For example, when one camera captures a narrow range for a long period of time, an obtained image has a high resolution. On the other hand, when another camera captures a wide range for a short period of time, an obtained image has a low resolution. In this way, there is a trade-off relationship between capturing time and resolution. Then, cameras are placed at the same location (location of eyepoint), the camera that capture a narrow range and the camera that captures a wide range are separated, and then the captured images are synthesized. As a result, an image captured at a high resolution in a wide range may be obtained, so the trade-off relationship between time and resolution is eliminated.
When a plurality of images are synthesized, a following image processing apparatus 110 is used. FIG. 28 shows an example of the configuration of the existing image processing apparatus 110. The image processing apparatus 110 includes a low-frequency extracting unit (LPF: Low Pass Filter) 111. The low-frequency extracting unit 111 inputs a high-resolution image 121, generated by a first camera that captures a subject at a narrow angle of view, and extracts a low-frequency image 122 of which the frequency is lower than a predetermined frequency band. In FIG. 28, together with processing blocks, histograms in which the abscissa axis represents a frequency and the ordinate axis represents a gain of a frequency are shown in regard to the high-resolution image 121 and the low-frequency image 122.
In addition, the image processing apparatus 110 includes a learning unit 112 and a parameter setting unit 113. The learning unit 112 learns a correspondence relationship of the high-resolution image 121 with respect to the low-frequency image 122 on the basis of the input high-resolution image 121 and low-frequency image 122. The parameter setting unit 113 sets various parameters. Then, the image processing apparatus 110 includes a mapping unit 114. The mapping unit 114 superimposes the high-resolution image 121, supplied from the parameter setting unit 113, on a low-resolution image 123 captured at a wide angle of view by a second camera (not shown) to generate a synthesized image 124. The mapping unit 114 outputs the synthesized image 124 to an external output device.
FIG. 29 shows an example of existing image processing executed by the image processing apparatus 110. First, a low-frequency extracting unit 111 reduces the band of the high-resolution image 121, captured at a narrow angle of view, to the low frequency of the low-resolution image 123 captured at a wide angle of view, and extracts the low-frequency image 122 (step S101). Subsequently, the learning unit 112 performs learning between the high-resolution image 121 and the low-frequency image 122 extracted by the low-frequency extracting unit 111, and the parameter setting unit 113 obtains parameters (step S102).
The mapping unit 114 uses the parameters set by the parameter setting unit 113, and performs mapping in which the high-resolution image 121 is superimposed at a position corresponding to the low-resolution image 123 in a wide angle of view (step S103). An image that is obtained by mapping the high-resolution image 121 on the low-resolution image 123 is output as a synthesized image.
Japanese Unexamined Patent Application Publication No. 9-139878 describes a technique for imaging with multiple foci. The above technique is intended to obtain an image of which the correct focus is achieved for any one of a long-distance view and a short-distance view. Then, the lenses of a plurality of cameras each include an outer lens that brings the long-distance view into focus and a center lens that brings the short-distance view into focus.